ACM international workshop on very%2Dlarge%2Dscale multimedia corpus

Towards multimodal emotion recognition a new approach.

Advances in Multimedia Modeling
Abstract. Attention is a psychological measurement of human reflection against stimulus. We propose a general framework of highlight detection by comparing attention intensity during the watching of sports videos. Three steps are involved: adaptive selection on salient features, unified attention estimation and highlight identification. Adaptive selection computes feature correlation to decide an optimal set of salient features. Unified estimation combines these features by the technique of multi-resolution auto-regressive (MAR) and thus creates a temporal curve of attention intensity. We rank the intensity of attention to discriminate boundaries of highlights. Such a framework alleviates semantic uncertainty around sport highlights and leads to an efficient and effective highlight detection. The advantages are as follows: (1) the capability of using data at coarse temporal resolutions; (2) the robustness against noise caused by modality asynchronism, perception uncertainty and feature mismatch; (3) the employment of Markovian constrains on content presentation, and (4) multi-resolution estimation on attention intensity, which enables the precise allocation of event boundaries. Key words:highlight detection, attention computation, sports video analysis 1
Bimodal Emotion Recognition.
Recent technological advances have enabled human users to interact with computers in ways previously unimaginable. Beyond the confines of the keyboard and mouse, new modalities for human-computer interaction such as voice, gesture, and force-feedback are emerging. Despite important advances, one necessary ingredient for natural interaction is still missing- emotions. This paper describes the challenging problem of bimodal emotion recognition and advocates the use of probabilistic graphical models when fusing the different modalities. We test our audio-visual emotion recognition approach on 38 subjects with 11 HCI-related affect states. The experimental results show that the average persondependent emotion recognition accuracy is greatly improved when both visual and audio information is used in classification.
Advances in Multimedia Modeling
Abstract. Attention is a psychological measurement of human reflection against stimulus. We propose a general framework of highlight detection by comparing attention intensity during the watching of sports videos. Three steps are involved: adaptive selection on salient features, unified attention estimation and highlight identification. Adaptive selection computes feature correlation to decide an optimal set of salient features. Unified estimation combines these features by the technique of multi-resolution auto-regressive (MAR) and thus creates a temporal curve of attention intensity. We rank the intensity of attention to discriminate boundaries of highlights. Such a framework alleviates semantic uncertainty around sport highlights and leads to an efficient and effective highlight detection. The advantages are as follows: (1) the capability of using data at coarse temporal resolutions; (2) the robustness against noise caused by modality asynchronism, perception uncertainty and feature mismatch; (3) the employment of Markovian constrains on content presentation, and (4) multi-resolution estimation on attention intensity, which enables the precise allocation of event boundaries. Key words:highlight detection, attention computation, sports video analysis 1
ACM SIGMM the first workshop on web%2Dscale multimedia corpus %28WSMC09%29.

Integrating Image Segmentation and Classification for Fuzzy Knowledge%2DBased Multimedia Indexing.
Abstract. In this paper we propose a methodology for semantic indexing of images, based on techniques of image segmentation, classification and fuzzy reasoning. The proposed knowledge-assisted analysis architecture integrates algorithms applied on three overlapping levels of semantic information: i) no semantics, i.e. segmentation based on low-level features such as color and shape, ii) mid-level semantics, such as concurrent image segmentation and object detection, region-based classification and, iii) rich semantics, i.e. fuzzy reasoning for extraction of implicit knowledge. In that way, we extract semantic description of raw multimedia content and use it for indexing and retrieval purposes, backed up by a fuzzy knowledge repository. We conducted several experiments to evaluate each technique, as well as the whole methodology in overall and, results show the potential of our approach. 1

Evidence Theory%2DBased Multimodal Emotion Recognition.


Towards Fully Automatic Image Segmentation Evaluation.

Spatiotemporal modeling and matching of video shots.

A multimodal approach to music transcription.

A multimodal approach to music transcription.

Low%2Dlevel feature fusion models for soccer scene classification.

Spatiotemporal semantic video segmentation.
Abstract—In this paper, we propose a framework to extend semantic labeling of images to video shot sequences and achieve efficient and semantic-aware spatiotemporal video segmentation. This task faces two major challenges, namely the temporal variations within a video sequence which affect image segmentation and labeling, and the computational cost of region labeling. Guided by these limitations, we design a method where spatiotemporal segmentation and object labeling are coupled to achieve semantic annotation of video shots. An internal graph structure that describes both visual and semantic properties of image and video regions is adopted. The process of spatiotemporal semantic segmentation is subdivided in two stages: Firstly, the video shot is split into small block of frames. Spatiotemporal regions (volumes) are extracted and labeled individually within each block. Then, we iteratively merge consecutive blocks by a matching procedure which considers both semantic and visual properties. Results on real video sequences show the potential of our approach. I.
Perplexity%2Dbased evidential neural network classifier fusion using mpeg%2D7 low%2Dlevel visual features.

K%2DSpace at TRECVid 2008.
Enlighten – Research publications by members of the University of Glasgow
Multi%2Dlevel Fusion for Semantic Video Content Indexing and Retrieval.

Analysis of vector space model and spatiotemporal segmentation for video indexing and retrieval.

ALICIA.
Spanish verbs display two past-tense forms, the pretérito and the imperfecto. We offer an account of the semantics of these forms within a situation semantics, addressing a number of theoretically interesting questions about how to realize a semantics for tense and events in that type of framework. We argue that each of these forms is unambiguous, and that the apparent variety of readings attested for them derives from interaction with other factors in the course of interpretation. The meaning of the imperfecto is constrained to always reflect atelic aktionsart. In addition, it contains a modal element, and a contextually-given accessibility relation over situations constrains the interpretation of the modal in ways that give rise to all the attested readings. The pretérito is indeterminate with respect to aktionsart, neither telic nor atelic. One or the other aktionsart may be forced by other factors in the clause in which the pretérito occurs, as well as by pragmatic contrast with the possibility of using the imperfecto. 1.

Performance Analysis of Multiple Classifier Fusion for Semantic Video Content Indexing and Retrieval.


SAMMI.
The SAMMIE 1 system is an in-car multimodal dialogue system for an MP3 application. It is used as a testing environment for our research in natural, intuitive mixed-initiative interaction, with particular emphasis on multimodal output planning and realization aimed to produce output adapted to the context, including the driver’s attention state w.r.t. the primary driving task. 1
Eurecom at TRECVid 2007 Extraction of High%2Dlevel Features.
In this paper we describe our experiments for the high level features extraction task of TRECVid 2007. Our approach is different than previous submissions in that we have implemented a multi-descriptors system. Five (5) experimentations are submitted based on: • Run 1: MPEG-7 global descriptors, • Run 2: MPEG-7 global and GET audio descriptors, • Run 3: MPEG-7 region descriptors using region based automatic segmentation method RBAS (A region merging approach incorporating geometric properties), • Run 4: Color and texture descriptors are extracted using three segmentation methods (A fixed image grid, watersheds and a technique based on minimum spanning trees MST), • Run 5: Combination of global and regions descriptors. The experimental results show that the performance can be improved with suitable concept models. Secondly, using audio features did not lead to performance improvement in our experiments.
K%2DSpace at TRECVID 2007.
In this paper we describe K-Space participation in TRECVid 2007. K-Space participated in two tasks, highlevel feature extraction and interactive search. We present our approaches for each of these activities and provide a brief analysis of our results. Our high-level feature submission utilized multi-modal low-level features which included visual, audio and temporal elements. Specific concept detectors (such as Face detectors) developed by K-Space partners were also used. We experimented with different machine learning approaches including logistic regression and support vector machines (SVM). Finally we also experimented with both early and late fusion for feature combination. This year we also participated in interactive search, submitting
A New Approach to Probabilistic Image Modeling with Multidimensional Hidden Markov Models.
Abstract. This paper presents a novel multi-dimensional hidden Markov model approach to tackle the complex issue of image modeling. We propose a set of efficient algorithms that avoids the exponential complexity of regular multidimensional HMMs for the most frequent algorithms (Baum-Welch and Viterbi) due to the use of a random dependency tree (DT-HMM). We provide the theoretical basis for these algorithms, and we show that their complexity remains as small as in the uni-dimensional case. A number of possible applications are given to illustrate the genericity of the approach. Experimental results are also presented in order to demonstrate the potential of the proposed DT-HMM for common image analysis tasks such as object segmentation, and tracking. 1
Classifier Fusion Combination Methods For Semantic Indexing in Video Content.

Graph%2DBased Spatio%2Dtemporal Region Extraction.

Personalization of hyperlinked video in interactive television.
The GMF4iTV project is building an end-to-end broadcast system for providing interactivity of TV programs through active video objects. This paper describes the personalization scheme that has been defined and implemented in this project. On the Authoring side, an interface tool allows the video producers to annotate both objects and associated additional content, with attributes from different ontologies. On the receiver side, a personalization engine compares these attributes with the user profile to personalize the interaction with the user. We describe the various components of the system and provide some application examples.
Hyperlinked Video with Moving Objects in Digital Television.

Improved Video Content Indexing by Multiple Latent Semantic Analysis.
Low-level features are now becoming insufficient to build efficient content-based retrieval systems. Users are not interested any longer in retrieving visually similar content, but they expect retrieval systems to also find documents with similar semantic content. Bridging the gap between low-level features and semantic content is a challenging task necessary for future retrieval systems. Latent Semantic Analysis (LSA) was successfully introduced to efficiently index text documents by detecting synonyms and the polysemy of words. We have successfully proposed an adaptation of LSA to model video content for object retrieval and semantic content estimation. Following this idea we now present a new model composed of multiple LSA&#039;s (M-LSA) to better represent the video content. In the experimental section, we make a comparison of LSA and M-LSA on two problems, namely object retrieval and semantic content estimation.
Using Structure for Video Object Retrieval.
The work presented in this paper aims at reducing the semantic gap  between low level video features and semantic video objects. The proposed method  for finding associations between segmented frame region characteristics relies on  the strength of Latent Semantic Analysis (LSA). Our previous experiments [1],  using color histograms and Gabor features, have rapidly shown the potential of  this approach but also uncovered some of its limitation. The use of structural  information is necessary, yet rarely employed for such a task. In this paper we  address two important issues. The first is to verify that using structural information  does indeed improve performance, while the second concerns the manner in  which this additional information is integrated within the framework. Here, we  propose two methods using the structural information. The first adds structural  constraints indirectly to the LSA during the preprocessing of the video, while the  other includes the structure directly within the LSA. Moreover, we will demonstrate  that when the structure is added directly to the LSA the performance gain  of combining visual (low level) and structural information is convincing.
Enhancing latent semantic analysis video object retrieval with structural information.
The work presented in this paper aims at reducing the semantic gap between low level video features and semantic video objects. The proposed method for finding associations between segmented frame region characteristics relies on the strength of Latent Semantic Analysis. Our previous experiments [1] have shown the potential of this approach but also uncovered some of its limitation. Here, we will present a method using the structural information within an LSA framework. Moreover, we will demonstrate the performance gain of combining visual (low level) and structural information.
Latent semantic indexing for semantic content detection of video shots.
Low-level features are now becoming insufficient to build efficient content-based retrieval systems. The interest of users is not anymore to retrieve visually similar content, but they expect that retrieval systems find documents with similar semantic content. Bridging the gap between low-level features and semantic content is a challenging task necessary for future retrieval systems. Latent Semantic Indexing (LSI) was successfully introduced to efficiently index text documents. In this paper we propose to adapt this technique to efficiently represent the visual content of video shots for semantic content detection. Although we restrict our approach to visual features, it can be extended with minor changes to audio and motion features to build a multi-modal system. The semantic content is then detected thanks to two classifiers: k-nearest neighbors and neural network classifiers. Finally, in the experimental section we show the performances of each classifier and the performance gain obtained with LSI features compared to traditional features.
Latent semantic indexing for semantic content detection of video shots.
Low-level features are now becoming insufficient to build efficient content-based retrieval systems. The interest of users is not anymore to retrieve visually similar content, but they expect that retrieval systems find documents with similar semantic content. Bridging the gap between low-level features and semantic content is a challenging task necessary for future retrieval systems. Latent Semantic Indexing (LSI) was successfully introduced to efficiently index text documents. In this paper we propose to adapt this technique to efficiently represent the visual content of video shots for semantic content detection. Although we restrict our approach to visual features, it can be extended with minor changes to audio and motion features to build a multi-modal system. The semantic content is then detected thanks to two classifiers: k-nearest neighbors and neural network classifiers. Finally, in the experimental section we show the performances of each classifier and the performance gain obtained with LSI features compared to traditional features.
Latent semantic analysis for an effective region%2Dbased video shot retrieval system.
We present a complete and e#cient framework for video shot indexing and retrieval. Video shots are described by their key-frame, themselves described by their regions. Regionbased approaches su#er from the complexity of segmentation and comparison tasks. A compact region-based shot representation is usually obtained thanks to vector-quantization method. We thus introduce LSA to reduce the noise inherent to the segmentation and the quantization processes. Then to better capture the content of video shots, we propose two original methods. The first takes advantage of a multi-scale segmentation of frames while the second uses multiple frames to represent a shot. Both approaches require more computation time during the pre-processing but not for indexing and comparison tasks. Indeed the extra information is included in the original signatures of shots. Finally we introduce a relevance feedback loop to optimize the search and propose a new method to optimize the e#ect of LSA. In the experimental section, we make an evaluation of latent semantic analysis and proposed approaches on two problems, namely object retrieval and semantic content estimation.
Generating TV summaries for CE%2Ddevices.

Semantic Feature Extraction using Mpeg Macro%2Dblock Classification.
In this paper, we present some first results in the extraction of semantic features from video sequences. Our approach is based on the classification of Mpeg DCT macro-blocks. Although it is clear that using macro-blocks imposes severe restrictions on the analysis accuracy of the image, it has the advantage of avoiding the complete decoding of the Mpeg stream. Our objective is to evaluate the quality of the Semantic Feature Extraction that can be obtained with this direct approach, to serve as a comparative baseline with more elaborate approaches.
Relational object recognition from large structural libraries.

Relational skeletons for retrieval in patent drawings.
This paper presents the evaluation of a number of algorithm alternatives for content based retrieval from a database of technical drawings representing patents. The objective is to help patent evaluators in their quest for a possible patent bearing too much similarity with the one under investigation. To achieve this, we have devised a system where images (drawings) are represented using attributed graphs based on the extracted line-patterns or histograms of attributes computed from the graphs. Retrieval is either performed using histogram comparison [6] or thanks to a graph similarity measure [9]. Promising results are presented along with possible work extention.
Generating Summaries Of Multi%2DEpisode Video.
The growing availability of multimedia data such as video on personal computers and home equipment creates a strong requirement for efficient tools to manipulate this type of data. Automatic summarization is one of these tools, which automatically creates a short version or subset of key-frames which contains as much information as possible as in the original video. Several approaches have been proposed to define and identify what is the most important content in a video. In this paper
Generating Summaries Of Multi%2DEpisode Video.
The growing availability of multimedia data such as video on personal computers and home equipment creates a strong requirement for efficient tools to manipulate this type of data. Automatic summarization is one of these tools, which automatically creates a short version or subset of key-frames which contains as much information as possible as in the original video. Several approaches have been proposed to define and identify what is the most important content in a video. In this paper
Sensitivity Analysis for Object Recognition from Large Structural Libraries.

Shape Retrieval by Inexact Graph Matching.
This paper describes a graph-matching technique for recognising line-pattern shapes in large image databases. The methodological contribution of the paper is to develop a Bayesian matching algorithm that uses edge-consistency and node attribute similarity. This information is used to determine the a posteriori probability of a query graph for each of the candidate matches in the database. The node feature-vectors are constructed by computing normalised histograms of pairwise geometric attributes. Attribute similarity is assessed by computing the Bhattacharyya distance between the histograms. Recognition is realisedbyselecting the candidate from the database which has the largest a posteriori probability. We illustrate the recognition technique on a database containing 2500 line patterns extracted from real-world imagery. Here the recognition technique is shown to significantly outperform a number of algorithm alternatives.
Structural Sensivity for Large%2DScale Line%2DPattern Recognition.

Line Pattern Retrieval Using Relational Histograms.
AbstractÐThis paper presents a new compact shape representation for retrieving line-patterns from large databases. The basic idea is to exploit both geometric attributes and structural information to construct a shape histogram. We realize this goal by computing the N-nearest neighbor graph for the lines-segments for each pattern. The edges of the neighborhood graphs are used to gate contributions to a two-dimensional pairwise geometric histogram. Shapes are indexed by searching for the line-pattern that maximizes the cross correlation of the normalized histogram bin-contents. We evaluate the new method on a database containing over 2,500 line-patterns each composed of hundreds of lines. Index TermsÐImage database, line patterns, content-based retrieval, relational representation, geometric features, histogram comparison. 1
Shape recognition from large image libraries by inexact graph matching.

Increased Extent of Characteristic Views using Shape%2Dfrom%2DShading for Object Recognition.
This paper investigates the use of shape-from-shading for object recognition. The local surface orientation information recovered using shape-fromshading is shown to provide useful input to an appearance-based object recognition scheme. We consider two representations which may be recovered from shading information - the needle-map, and the local curvature shapeindex - and examine their relative performance for object recognition. Specifically, we use a histogram-comparison technique, and focus upon the relative stability of the representations to small changes of viewpoint. We demonstrate that the needle-map representation allows the view-sphere to be spanned using a significantly smaller number of characteristic views than using either the raw images or the shape index. 1 Introduction  Despite long-term interest in shape-from-hading (SFS), and psychophysical evidence that it is a key process in 3D surface perception [15], there are few reports of its use in practical object-recogniti...
Fuzzy Relational Distance for Large%2DScale Object Recognition.
This paper presents a new similarity measure for object recognition from large libraries of line-patterns. The measure draws its inspiration from both the Hausdorff distance and a recently reported Bayesian consistency measure that has been sucessfully used for graphbased correspondence matching. The measure uses robust error-kernels to gauge the similarity of pairwise attribute relations defined on the edges of nearest neighbour graphs. We use the similarity measure in a recognition experiment which involves a library of over 1000 line-patterns. A sensitivity study reveals that the method is capable of delivering a recognition accuracy of 98%. A comparative study reveals that the method is most effective when a Gaussian kernel or Huber&#039;s robust kernel is used to weight the attribute relations. Moreover, the method consistently outperforms Rucklidge &#039;s median Hausdorff distance.  1 Introduction  Object recognition from large libraries of images holds the key to the automatic manipulati...
Relational Histograms for Shape Indexing.
This paper is concerned with the retrieval of images from large databases based on their shape similarity to a query image. Our approach is based on two dimensional histograms that encode both the local and global geometric properties of the shapes. The pairwise attributes are the directed segment relative angle and directed relative position. The novelty of the proposed approach is to simultaneously use the relational and structural constraints, derived from an adjacency graph, to gate histogram contributions. We investiguate the retrieval capabilities of the method for various queries. We also investigate the robustness of the method to segmentation errors. We conclude that a relational histogram of pairwise segment attributes presents a very efficient way of indexing into large databases. The optimal configuration is obtained when the local features are constructed from six neighbouring segments pairs. Moreover, a sensitivity analysis reveals that segmentation errors do not affect t...
Graph Matching for Shape Retrieval.
This paper describes a Bayesian graph matching algorithm for  data-mining from large structural data-bases. The matching algorithm  uses edge-consistency and node attribute similaritytodetermine  the aposteriori probability of a query graph for eachofthe  candidate matches in the data-base. The node feature-vectors are  constructed by computing normalised histograms of pairwise geometric  attributes. Attribute similarity is assessed by computing  the Bhattacharyya distance between the histograms. Recognition  is realised by selecting the candidate from the data-base which has  the largest a posteriori probability. We illustrate the recognition  technique on a data-base containing 2500 line patterns extracted  from real-world imagery. Here the recognition technique is shown  to significantly outperform a number of algorithm alternatives.  1 Introduction  Since BarrowandPopplestone [1] first suggested that relational structures could be used to representandinterpret 2D scenes, there has ...
Object Recognition from Large Structural Libraries.
. This paper presents a new similarity measure for object recognition from large libraries of line-patterns. The measure commences from a Bayesian consistency criterion which as been developed for locating correspondence matches between attributed relational graphs using iterative relaxation operations. The aim in this paper is to simplify the consistency measure so that it may used in a non-iterative manner without the need to compute explicit correspondence matches. This considerably reduces the computational overheads and renders the consistency measure suitable for large-scale object recognition. The measure uses robust error-kernels to gauge the similarity of pairwise attribute relations defined on the edges of nearest neighbour graphs. We use the similarity measure in a recognition experiment which involves a library of over 2000 line-patterns. A sensitivity study reveals that the method is capable of delivering a recognition accuracy of 94%. A comparative study reveals that the ...
Structurally Gated Pairwise Geometric Histograms for Shape Indexing.

